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In this paper we discuss the use of ontologies and their applications to assessment. 
The work reported herein is part of parallel efforts to explore the feasibility and utility 
of ontologies. The first application discussed is embedded in a larger research program 
to develop assessment models and tools for Naval distributed learning. CRESST is 
under contract to the Office of Naval Research and the first application of our work is to 
U.S. Marine Corps (USMC) marksmanship training. Our USMC work is focused on 
developing online assessments of Marines' knowledge of rifle marksmanship. The 
second application discussed is embedded in a larger research program to develop 
online assessment design and delivery tools for middle school science teachers. 

In each case, ontologies are used as a computational structure to capture explicit 
representations of knowledge, whether the knowledge is content-focused (e.g., rifle 
marksmanship or physics), or assessment design-focused (e.g., assessment design 
principles). In both cases ontologies are key components in the development, delivery, 
design, and scoring of assessments. 



Definition of an Ontology 

An ontology provides a shared and common understanding of a domain that can 
be communicated among people and computational systems (Fensel, Wahlster, 
Lieberman, & Hendler, 2003). The ontology captures one or more experts' conceptual 
representation of a domain expressed in terms of concepts and the relationships among 
the concepts. An ontology is a commitment to a point of view of how a domain is 
structured but there can be multiple representations (Chandrasekaran, Josephson, & 
Benjamins, 1999; McGuinness, 2003). Ontologies are important because they provide a 
common, explicit framework for sharing and using knowledge. More concretely, an 
ontology standardizes the terms and structure of the domain. The standardization 
makes possible sharing of the ontology-and thus the knowledge contained therein-for 
use across multiple computer platforms for different applications (Gruber, 1995). 



1 The work reported herein was supported under the Educational Research and Development Centers 
Program, PR/Award Number R305B960002, as administered by the Office of Educational Research and 
Improvement, U.S. Department of Education, Office of Naval Research Award Number N00014-02-1- 
0179, as administered by the Office of Naval Research, and the National Science Foundation Award 
Number REC-0129406. The findings and opinions expressed in this report do not reflect the positions or 
policies of the National Institute on Student Achievement, Curriculum, and Assessment, the Office of 
Educational Research and Improvement, the U.S. Department of Education, the Office of Naval Research, 
or the National Science Foundation. 
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Ontologies were first developed as part of the AI research effort to facilitate knowledge 
sharing and reuse. The use of ontologies has extended recently to fields such as 
information retrieval, knowledge management, medical guidelines, military, and e- 
commerce. CRESST is now applying ontologies to assessment. 

Ontologies to Support Assessment and Instruction 

For assessment and instructional purposes, the capability to express the concepts 
in a domain, the links among the concepts, and the governing constraints offers clear 
advantages over relational or highly structured data models. It is often the case that the 
representation of a domain is best represented as network (vs. a strictly hierarchical 
representation, for example), especially in knowledge-rich applications. 

The existence of computational tools to create, edit, maintain, and exchange 
ontologies makes feasible the use of ontologies in assessment and instruction. For 
example, Protege, originally developed in 1987 at Stanford University, is in its third 
generation (Gennari et al., 2002). Protege has an easy-to-use graphical user interface, 
Java implementation, and an active developer community. Similar products are 
available from both academic and commercial vendors. 

At CRESST, we are implementing ontologies for a variety of applications related to 
assessment: 



• To capture the structure of a domain 

• To capture experts' representation of a domain 

• To encode and bind content to a domain structure 

• To score knowledge maps 

• To package and deliver content at different grain sizes 

• To be part of a recommender system 

• To provide a structure to guide the automated design of assessments 

In addition, we are evaluating the potential of ontologies to assist in the creation of 
assessments on-the-fly (e.g., dynamic generation of terms and links for the CRESST 
knowledge mapper depending on the current state of the examinee's knowledge). 



In the following sections, we present two examples of how we are using ontologies 
for assessment purposes. In the first example we describe an application for the domain 
of rifle marksmanship. In the second example, we discuss the use of ontologies in the 
design of authoring systems to link assessment, content, and cognitive demands. We 
believe ontologies and assessment objects support the rapid development of an 
assessment authoring system and ultimately, through use of the authoring system, 
result in higher quality assessments developed by end-users (e.g., teachers). 

Example 1: Ontology of USMC Rifle Marksmanship Knowledge 

We have developed an ontology of rifle marksmanship as part of our work with 
the U.S. Marine Corps. One aspect of this work was to capture the knowledge related to 
rifle marksmanship. The overall purpose for developing an ontology was to capture the 
knowledge and structure of the domain in a way that would allow us to explore the use 
of ontologies for assessment and instructional purposes. We judged the domain of rifle 
marksmanship to be an ideal candidate to represent in an ontology because the domain 
is bounded and there was agreement among the domain experts about the important 
topics. A secondary purpose was packaging the knowledge for potential use by the 
Marine Corps training command. 

Our research questions with respect to ontologies are related to investigating how 
ontologies can be used to support (a) the automated scoring of knowledge maps, (b) the 
conceptual searching of information, and (c) given performance information on 
different assessments, the generation of individualized recommendations of relevant 
information that could fill in an individual's knowledge gaps. 

Domain Structure 

Our knowledge engineering strategy was to capture knowledge in two 
representations: (a) as outlined by doctrine (e.g., USMC field manuals), information 
which could be organized as a hierarchically structured body of knowledge; and (b) as 
perceived by experts (e.g., coaches, snipers, rifle team members), information which 
could be organized conceptually (i.e., as a network) to reflect how domain experts 
perceived the knowledge to be interrelated. 

Currently, our rifle marksmanship ontology contains 168 different concepts that 
cover seven fundamentals of rifle marksmanship and 160 relationships among the 
concepts using 16 relationship types. Figure 1 shows a portion of the hierarchy of the 
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ontology. The structure of the content is captured by the Knowledge class. The 
hierarchical structure shows the taxonomy of class and subclass relationships among 
the topics. 

9~© Knowledge* 

9 - ©Procedural* 

—© ProceduralMetaClass M 
9 -© FundamentalsOfMarksmanship* M 
9~©AimingProcess 

— © EyeRelief 

— © EyeOnFrontSightPost 
.— © SightAlignment 

— © SightPicture 

-© NaturalPointOfAim M 
<?-©Accuracy M 
L-© CenterMass 
-©Follow-Through 
-©Recovery 
9~© BreathControl M 
| L ~© Matur alRespiratoryPause 

o © Tri g g e rC 0 ntroi^ 1 ! 

9-© ElementsOfAGoodShootingPosition M 
-© BoneSupport 
■ -© NaturalPointOfAim M 

-© MuscularRelaxationLoopSling M 
*-© MucularTensionHastySling M 
9 -© 7FactorsCommonToAIIShootingPositions 

- © StockWeldPlacement 

- © ForwardHandPlacement 

- © PlacementOfButtstocklnShoulder 

- © ElbowPlacement 

- © MuscularRelaxationLoopSling M 
-© MucularTensionHastySling M 

©- © BreathControl M 
9- © T riggerControl M 
9-© FingerPlacement 

L- © ResettingTheTrigger 
j — (©) TriggerSqueeze 
h©GnpOfFiringHand M 
© UninterruptedTC 

La i'T/i Intom into rlTC- 

Figure 1. Example of the rifle marksmanship content 
organized hierarchically. 

Figure 2 shows how the content is organized as perceived by our domain experts. 
In this case, the organization is a network and represented by the Relationship class. The 
Relationship class is made up of subclasses that represent high-level relation types (e.g., 
causal, part/ whole). Subclasses of each relation type represent increasingly specific 
relations (e.g., PartOf is a particular kind of relation within the PartWhole class). The 
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right pane in Figure 2 shows specific instances of the PartOf relation that directly 
connect different topics shown in Figure 1. A visual representation of the set of relations 
is shown in Figure 3. Our assumption is that the hierarchical representation reflects the 
organizational structure of the content similar to a table of contents, and the relational 
structure captures the detailed relations that presumably underlie deep understanding 
of the content. The significance of this distinction will become apparent when we 
describe how the ontology is used for scoring knowledge maps, semantic searching, 
and recommending relevant content. 
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© Instruction* 
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I <|> BattlesightZero~>DatabooiT 



| <§> BattlesightZero->Zero 
<t> Call->Databook 
<$> Carries->WeaponHandling 
| <|> CenterlineOfTheBore-->Zeroing 
1 <|> Databook~>FundamentalsOfMarksmanship 
© EyeOnFrontSightPost->SightPicture 
I <&>LineOfSight-->Zeroing 
<£>NaturalPointOfAim~>AimingProcess 
©NaturalPointOfAim~>FundamentalsOfMarksmans 
<S>NaturalFtespiratoryPause~>BreathControl 
| <|> Plot->Databook 
© Point OfAim-->Zeroing 
© Range~>Zeroing 
<S> Ftemarks->Databook 
| <$> FtifleFiringPositions-->Follow-Through 
©FtifleFiringPositions-->RifleMarksmanship 
<S> SafetyRules->WeaponHandling 
<t> SightAlignment->AimingProcess 
© Sight Picture-->AimingProcess 
©Sight Picture-->FundamentalsOfMarksmanship 
<S> Trajectory~>Zeroing 
<S> Transports~>WeaponHandling 
<i> TrueZero->Databook 
<$> T rueZero->Zero 

<$>WeaponCondition->WeaponHandling 
4> WindCalls-->Databook 
©WindVelocrty->EffectsOfWeather 
<|>Zero->Zeroing 



Figure 2. Example of relationship classes. The relationship class specifies how the content is related 
conceptually. 
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Figure 3. Composite expert map that is captured in the ontology Relationship class. 



Binding Content to the Ontology Structure 

Many ontologies typically capture only the structure of the domain (e.g., Figure 1). 
However, to be useful instructionally, content would ideally be bound to the structure. 
For example, Figure 4 shows an example of how content is related directly to objects in 
the ontology. For each topic, we have defined different knowledge types — conceptual 
(or declarative) knowledge and procedural knowledge. Further, we have partitioned 
the information into subtypes: definition, explanation (i.e., why the topic is important), 
and elaboration (i.e., supplemental information). Although not shown in Figure 4, we 
have also allowed for the inclusion of different media types (e.g., video, picture, URL). 
For example, for the topic BreathControl we have a video demonstrating the effects of 
breathing on the position of the rifle muzzle and bullet strike (breathing causes the rifle 
to move vertically; firing while breathing results in a vertical dispersion of shots). 



Source material was drawn from the U.S. Marine Corps rifle marksmanship 
manual (USMC, 2001). Marksmanship training is derived from this manual. For 
concepts, the instructional content is delineated in terms of definition, explanation, 
elaboration, and multimedia examples (e.g., a picture of the trigger) where appropriate. 
For relations, the instructional content was an explanation of why the particular relation 



holds under the particular conditions. 
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TriggerControl 



Conceptual Definition 



The skillful manipulation of the trigger that 
causes the rifle to fire without disturbing 
sight alignment or sight picture. 



Conceptual Explanation 



Controlling the trigger is a mental 
process, while pulling the trigger is a 
physical process. 

Resetting the trigger places the trigger in 
position to fire the next shot without 
havi ng to re e stablish trig ger fing er 






Conceptual Elaboration 



The trigger finger should contact the 
trigger naturally. The trigger finger 
should not contact the rifle receiver or 
trigger guard. 



Procedure 



After obtaining sight picture, the Marine |£ 
applies smooth, continuous pressure 
rearward on the trigger until the shot is 
fired. 

INTERRUPTED TRIGGER CONTROL: 



Procedural Explanation 



Procedural Elaboration 



Figure 4. Example of the rifle marksmanship content bound to the topic TriggerControl. 
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Ontology Application: Automated Scoring of Knowledge Maps 

We have developed automated methods of scoring knowledge maps in our prior 
work based on comparing students' knowledge maps to a criterion map (e.g., Chung & 
Baker, 1997; Chung, Harmon, & Baker, 2001; Chung, O'Neil, & Herl, 1999; Herl, Baker, 
& Niemi, 1996; Herl, O'Neil, Chung, & Schacter, 1999; Klein, Chung, Osmundson, Herl, 
& O'Neil, 2002; Lee, 2000; Osmundson, Chung, Herl, & Klein, 1999). One shortcoming 
of this approach is that the score is a count (i.e., the number of propositions [node-link- 
node tuple] in the student's map that are also in the criterion map). An alternative 
approach is to score each proposition on a scale. This approach yields information on 
the degree of accuracy of the proposition (e.g., Osmundson et al., 1999; Ruiz-Primo, 
Schultz, Li, & Shavelson, 2001). 

Several studies have investigated the technical properties of scoring knowledge 
maps. For example, Ruiz-Primo et al. (2001), in addition to scoring knowledge maps 
proposition-by-proposition, also scored students' maps against a criterion map. The 
correlation between the proposition accuracy score and expert-based score was 
sufficiently high for Ruiz-Primo et al. to conclude that an expert-based method was the 
most efficient scoring method (i.e., in terms of scoring time and reliability of scores). 
Similar results were found by Osmundson et al. (1999) and Chung et al. (2001). 

The findings of Ruiz-Primo et al. (2001) are consistent with earlier work by Herl 
(1995), Herl et al. (1996), and Osmundson et al. (1999). In general, scoring student 
knowledge maps using expert-based referents has been found to discriminate between 
experts and novices (Herl, 1995; Herl et al., 1996), discriminate between different levels 
of student performance (Herl, 1995; Herl et al., 1996), relate moderately to external 
measures (Aguirre-Munoz, 2000; Herl, 1995; Herl et al., 1996; Klein, Chung, 
Osmundson, Herl, & O'Neil, 2002; Lee, 2000; Osmundson et al., 1999), detect changes in 
learning (Chung et al., 2001; Osmundson et al., 1999; Schacter, Herl, Chung, Dennis, & 
O'Neil, 1999), and be sensitive to language proficiency (Aguirre-Munoz, 2000; Lee, 
. 2000 ). 

Given the expressive potential of ontologies and the utility of knowledge mapping 
as an assessment task, one capability we are currently implementing is using the 
ontology as a domain template against which to score knowledge maps. Domain expert 
maps can easily be captured and folded into the ontology via the Relationship class 
(Figure 2). Similarly, propositions that are scored individually can also be folded in. By 
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integrating both methods, both the expressive power of the ontology and the scoring 
potential are increased. 

Ontology Application: Recommending Individualized Instructional Content 

Because of how we have structured the ontology (i.e., hierarchical and 
network/conceptual representations) and because we have bound content at different 
grain sizes to specific topics in the ontology, we now have the means to deliver content 
at different grain sizes depending on the application. In this section we describe our 
technique for identifying knowledge gaps and delivering individualized content. 

Identifying knowledge gaps. The first step in recommending individualized 
content is to identify an individual's knowledge gaps. Once the gaps are identified, 
relevant content needs to be retrieved and delivered to the individual. 

Identifying what students do and do not know is accomplished by diagnostic 
assessments. For example, our strategy for assessing Marines' understanding of rifle 
marksmanship is to use a range of measures that reflect different cognitive demands. 
For example, we broadly sample their knowledge of basic definitions of major ideas (via 
selected response matching), their cause-effect knowledge of how different aspects of 
shooting relate to each other causally (via constructed-response knowledge maps), their 
skill at diagnosing improper shooting positions (via QuickTime VR), and their 
capability to fix any improper positions identified (via QuickTime VR). 

This assessment information is then fused together using Bayesian networks to 
yield probabilities on the degree to which a Marine understands different topics of rifle 
marksmanship. A Bayesian inference network, also known as an influence or 
probabilistic causal network, depicts the causal structure of a phenomenon in terms of 
nodes and relations (Jensen, 2001). Nodes represent states, and links represent the 
influence relations among the nodes. Node states can be observable or unobservable. 

The utility of a Bayesian inference network is that it yields the probability that an 
unobservable variable is in a particular state (e.g., understands trigger control) given 
observable evidence (e.g., knowledge map score). The probability of the unobservable 
variable being in a particular state is the inference made about student understanding. 
Figure 5 shows the Bayesian network that depicts presumed knowledge dependencies. 
Observable nodes are omitted for clarity; however, in general, the leaf nodes would 
have observables linked to them. 
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in general be linked to the leaf nodes. 



Conceptual indexing. Once knowledge gaps are identified, content needs to be 
retrieved and delivered to the Marine. The conceptual representation in the ontology 
plays one key role in this process — we use the domain experts' representation to guide 
the selection of relevant information. For example, suppose a Marine's knowledge of 
TriggerControl is identified as poor based on the probabilities yielded from the Bayesian 
network. Content from the topic TriggerControl is retrieved, as well as the neighbors of 
TriggerControl (i.e., fan-in and fan-out nodes). In this way the conceptual representation 
is used to retrieve presumably the most relevant and contextualized information. 

The second key role is to use the performance on items of different cognitive 
demands to retrieve content at different grain sizes. Because the assessment items 
reflect different cognitive demands, presumably we have information on Marines' 
depth of understanding. Thus, we make use of the partitioning of content in each of the 
topics (see Figure 4) to deliver information at different grain sizes (e.g., definitions, 
explanations, elaborations, and so forth) depending on the Marine's performance on the 
different items. 

To provide a concrete example of how such an approach would work, we present 
an example derived from a Marine's performance on assessments of rifle 
marksmanship. The example is for the concept of TriggerControl (the skillful 
manipulation of the trigger that causes the rifle to fire without disturbing sight 
alignment). A particular student scored poorly on items that asked for (a) a simple 
definition of trigger control, (b) how trigger control relates to sight alignment, and (c) 
the pattern of shots for a shooter with poor trigger control. From this set of 
observations, one inference that could be drawn is that this student has little or no 
knowledge of trigger control. The instructional remediation for this student could be to 
provide information on (a) trigger control — definition, explanation, and elaboration; (b) 
how trigger control is related to sight alignment (e.g., "A firm grip helps maintain good 
sight alignment because the grip helps ensure that the trigger is pulled straight toward 
the rear of the rifle."); and (c) the shot-dispersion pattern for poor trigger control with a 
picture and explanatory information. 

Summary 

We have described several uses of ontologies for assessment and instructional 
purposes. The technology behind ontologies has matured to the point that 
computational tools are readily available. In the context of rifle marksmanship, we 
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described techniques for using ontologies for automated scoring of knowledge maps, 
conceptual searching, and individualized content delivery. 

Example 2: Authoring Support for Assessment Design 
Assessment Authoring Support 

The promise of an ontology is in the presumably accurate (but not necessarily 
consensus) representation of the assessment model — the coherent packaging of 
concepts and links. This representation can be used for example, to provide guidance to 
assessment authors as they design assessments for particular purposes under particular 
constraints. If the ontology describes the conditions under which an essay task 
possessing particular characteristics is appropriate, then an assessment authoring 
system can, at minimum, check that authors of this particular essay task use the task to 
measure the appropriate type of knowledge. 

An ontology that explicitly represents an assessment model implies a given 
structure and constraints (via the relations among concepts). For assessment authoring 
purposes, structure is of very high utility because it allows the enforcement of a 
common and consistent framework. This structure can be leveraged to assist assessment 
authors (particularly non-experts) in designing assessments. Assessment authoring 
support could be in the form of (a) aiding assessment authors to populate the 
assessment ontology with values specific to the users' purposes, and (b) system 
constraint checking that would ensure that assessment authors are alerted to 
incompatible values. 

In terms of aiding assessment authors, an authoring system can traverse the 
assessment ontology and gather information from users for only those parameters that 
matter. This structure enforces the specification of important information and ignores 
variables outside the ontology. This scenario assumes the ontology is reasonably 
complete and accurate. 

Constraint checking is carried out as the assessment author iteratively refines the 
values of the assessment parameters. An ontology network that converges to a steady- 
state condition implies that all constraints have been satisfied and all values for all 
assessment parameters are (simultaneously) acceptable. 

Additional authoring support could be provided by folding in assessment 
guideline information. Because of the flexibility of the ontology structure, slots could be 
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added to bind guideline information to specific assessment concepts. The purpose of 
providing assessment guideline information would be to bolster non-experts' 
(presumed) gaps in knowledge. Non-expert users are likely to have spotty knowledge 
of assessment in general — perhaps specific knowledge of only a few concepts and 
relations. Guidelines tied to concepts and relations should bolster what the assessment 
author knows about the domain, and just as important, what the person does not know. 
Similar applications of ontologies have been used to support physicians via clinical 
practice guidelines (Bernstam et al., 2000; de Clercq, Hasmon, Blom, & Korsten, 2001). 

In a specific application of this approach, funded by the Interagency Educational 
Research Initiative (IERI), we are building an online, research-based system to assist 
middle school science teachers to build better assessments of complex learning. Because 
many educational assessments and accountability systems reflect neither rigorous 
subject matter analyses nor adequate attention to the cognitive demands of the tasks 
used, the validity and credibility of their results are suspect (Linn, Baker, & Dunbar, 
1991). Our approach, model-based assessment, developed with OERI support (Baker, 
1997; Niemi, 1996), offers a tested alternative. Tests and assessments intended to 
provide information to key users are based on common representations of states of 
student understanding, the cognitive demands of instructional and assessment tasks, 
and the structure of knowledge and procedures defining a domain within a subject 
area. From a common blueprint, assessments can be developed and summarized at 
different levels of granularity appropriate for instructional diagnosis or for summative 
purposes. 

The Assessment Design and Delivery System (ADDS) we are building has three 
main features: (1) it provides utilities for individual teachers or teams of teachers to 
become designers and users of assessments that yield actionable information to guide 
their practice and student learning, (2) it embeds content, assessment, and pedagogical 
knowledge to assist teachers in both designing assessments and interpreting student 
progress, and (3) it produces valid results for classroom-based inferences with the 
potential for aggregation of results for policy uses. 

Our work to date has focused on the knowledge construction and design work 
necessary to build the system. The first step was to construct an ontology of organizing 
concepts and principles in a key domain of middle school physical science, force and 
motion. To construct this ontology we elicited major concepts and principles from a 
team of six physicists over a two-day period. Figure 6 shows a map of some of these 
"big ideas" and the relationships among them. Beginning the construction of a domain 
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is consistent with an extensive research base demonstrating that organization of 
knowledge typifies expert performance, and that domain knowledge, in the process of 
deepening understanding and developing greater expertise, becomes increasingly 
organized around abstractions such as concepts and principles (e.g., Bereiter & 
Scardamalia, 1986; Chi & Ceci, 1987; Chi, Glaser, & Rees, 1982; Glaser & Chi, 1988; 
Larkin, McDermott, Simon, & Simon, 1980; Niemi, 1996). Understanding of core 
principles and concepts results in more flexible and generalizable knowledge use, 
improves problem solving, and makes it easier to make sense of and master new facts 
and procedures (e.g., Ausubel, 1968; Chi & Ceci, 1987; Gelman & Lee Gattis, 1995; 
Larkin, McDermott, Simon, & Simon, 1980; Silver, 1981). 
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Figure 6. Part of an ontology for force and motion. 



The second step in building ADDS was to identify the types of problems and tasks 
students should be able to complete in order to demonstrate mastery of each major 
element of knowledge in the ontology, and to identify the cognitive demands associated 
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with each of those tasks. We have also developed an assessment database that holds 
both completed tasks and components of tasks such as information sources that may be 
textual, graphic, or multimedia. These information sources can be combined with other 
information sources and with questions or prompts to create new tasks. Tasks and task 
components in the assessment database are tagged with attributes that include: 
assessment purpose, type of assessment (explanation, problem solving, knowledge 
map), subject area and domain, relevant state standards, linguistic complexity, 
cognitive demands, development history, usage history, item quality, and scoring 
information. This tagging makes it possible to match tasks and components with 
elements in the domain ontology. 

In essence ADDS matches standards or topics in physics to a domain ontology that 
has embedded within it links to an array of reusable assessment components. Teachers 
engaged in the assessment design process can start either with standards (analyzed in 
terms of cognitive demands) or with topics and types of knowledge in physics. ADDS 
will then guide them to select or build appropriate types of assessment. Appendix A 
shows, the task builder screen that enables teachers to assemble task components, 
constrained by the standards or knowledge elements they are trying to assess. 
Ultimately ADDS will also contain a system for delivering and automatically scoring 
some assessments, and a reporting and interpretation tool to assist teachers in using the 
assessment information they obtain from the system. The reporting tool will use the 
Bayesian strategies sketched out earlier. 

The use of ontologies to support assessment authoring seems particularly 
promising because of the nature of the anticipated authors: non-experts who lack 
breadth and depth of knowledge of assessment. An ontology-based authoring system 
can impose structure on the authoring task as well as check that user-specified values 
simultaneously satisfy all constraints among the assessment components. 
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